Recently, several parametric techniques for the encoding of multi-channel/multi-object audio signals have been proposed. Each system has unique advantages and disadvantages w.r.t. its characteristics such as the type of parametric characterization, dependence/independence from a specific loudspeaker setup etc. Different parametric techniques are optimized for different encoding strategies.
As an example, the Directional Audio Coding (DirAC) format for the representation of multi-channel sound is based on a downmix signal and side information containing direction and diffuseness parameters for a number of frequency subbands. Due to this parametrization, the DirAC system can be used to easily implement e.g. directional filtering and in this way to isolate sound that originates from a particular direction relative to a microphone array used to pick up the sound. In this way, DirAC can also be regarded as an acoustic front-end that is capable of certain spatial processing.
As a further example, Spatial Audio Object Coding (SAOC) ISO/IEC, “MPEG audio technologies—Part. 2: Spatial Audio Object Coding (SAOC)”, ISO/IEC JTC1/SC29/WG11 (MPEG) FCD 23003-2, J. Herre, S. Disch, J. Hilpert, O. Hellmuth: “From SAC to SAOC—Recent Developments in Parametric Coding of Spatial Audio”, 22nd Regional UK AES Conference, Cambridge, UK, April 2007, J. Engdeg{dot over (a)}rd, B. Resch, C. Falch, O. Hellmuth, J. Hilpert, A. Hölzer, L. Terentiev, J. Breebaart, J. Koppens, E. Schuijers and W. Oomen: “Spatial Audio Object Coding (SAOC)—The Upcoming MPEG Standard on Parametric Object Based Audio Coding”, 124th AES. Convention, Amsterdam 2008, Preprint 7377, is a parametric coding system that represents audio scenes containing multiple audio objects in a bitrate-efficient way.
Here, the representation is based on a downmix signal and parametric side information. In contrast to DirAC, which aims at representing the original spatial sound scene as it was picked up by the microphone array, SAOC does not aim at reconstructing a natural sound scene. Instead, a number of audio objects (sound sources) are transmitted and are combined in an SAOC decoder into a target sound scene according to the preferences of the user at the decoder terminal, i.e. the user can freely and interactively position and manipulate each of the sound objects.
Generally, in multi-channel reproduction and listening, a listener is surrounded by multiple loudspeakers. Various methods exist to capture audio signals for specific setups. One general goal in the reproduction is to reproduce the spatial composition of an originally recorded signal, i.e. the origin of individual audio source, such as the location of a trumpet within an orchestra. Several loudspeaker setups are fairly common and can create different spatial impressions. Without using special post-production techniques, the commonly known two-channel stereo setups can only recreate auditory events on a line between the two loudspeakers. This is mainly achieved by so-called “amplitude-panning”, where the amplitude of the signal associated to one audio source is distributed between the two loudspeakers depending on the position of the audio source with respect to the loudspeakers. This is usually done during recording or subsequent mixing. That is, an audio source coming from the far-left with respect to the listening position will be mainly reproduced by the left loudspeaker, whereas an audio source in front of the listening position will be reproduced with identical amplitude (level) by both loudspeakers. However, sound emanating from other directions cannot be reproduced.
Consequently, by using more loudspeakers that are positioned around the listener, more directions can be covered and a more natural spatial impression can be created. The probably most well known multi-channel loudspeaker layout is the 5.1 standard (ITU-R775-1), which consists of 5 loudspeakers, whose azimuthal angles with respect to the listening position are predetermined to be 0°, ±30° and ±110°. That means, that during recording or mixing the signal is tailored to that specific loudspeaker configuration and deviations of a reproduction set up from the standard will result in decreased reproduction quality.
Numerous other systems with varying numbers of loudspeakers located at different directions have also been proposed. Professional systems, especially in theaters and sound installations, also include loudspeakers at different heights.
According to the different reproduction set-ups, several different recording methods have been designed and proposed for the previously mentioned loudspeaker systems, in order to record and reproduce the spatial impression in the listening situation as it would have been perceived in the recording environment. A theoretically ideal way of recording spatial sound for a chosen multi-channel loudspeaker system would be to use the same number of microphones as there are loudspeakers. In such a case, the directivity patterns of the microphones should also correspond to the loudspeaker layout, such that sound from any single direction would only be recorded with a small number of microphones (1, 2 or more). Each microphone is associated to a specific loudspeaker. The more loudspeakers used in reproduction, the narrower the directivity patterns of the microphones have to be. However, narrow directional microphones are rather expensive and typically have a non-flat frequency response, degrading the quality of the recorded sound in an undesirable manner. Furthermore, using several microphones with too broad directivity patterns as input to multi-channel reproduction results in a colored and blurred auditory perception due to the fact that sound emanating from a single direction would usually be reproduced with more loudspeakers than may be used as it would be recorded with microphones associated to different loudspeakers. Generally, currently available microphones are best suited for two-channel recordings and reproductions, that is, these are designed without the goal of a reproduction of a surrounding spatial impression.
From the point of view from microphone-design, several approaches have been discussed to adapt the directivity patterns of microphones to the demands in spatial-audio-reproduction. Generally, all microphones capture sound differently depending on the direction of arrival of the sound to the microphone. That is, microphones have a different sensitivity, depending on the direction of arrival of the recorded sound. In some microphones, this effect is minor, as they capture sound almost independently of the direction. These microphones are generally called omnidirectional microphones. In a typical microphone design, a secular diaphragm is attached to a small airtight enclosure. If the diaphragm is not attached to the enclosure and sound reaches it equally from each side, its directional pattern has two lobes. That is, such a microphone captures sound with equal sensitivity from both front and back of the diaphragm, however, with inverse polarities. Such a microphone does not capture sound coming from the direction coincident to the plane of the diaphragm, i.e. perpendicular to the direction of maximum sensitivity. Such a directional pattern is called dipole, or figure-of-eight.
Omnidirectional microphones may also be modified into directional microphones, using a non-airtight enclosure for the microphone. The enclosure is especially constructed such, that the sound waves are allowed to propagate through the enclosure and reach the diaphragm, wherein some directions of propagation are advantageous, such that the directional pattern of such a microphone becomes a pattern between omnidirectional and dipole. Those patterns may, for example, have two lobes. However, the lobes may have different strength. Some commonly known microphones have patterns that have only one single lobe. The most important example is the cardioid pattern, where the directional function D can be expressed as D=1+cos(θ), θ being the direction of arrival of sound. The directional function such quantifies, what fraction of incoming sound amplitude is captured, depending on different direction.
The previously discussed omnidirectional patterns are also called zeroeth-order patterns and the other patterns mentioned previously (dipole and cardioid) are called first-order patterns. All the previously discussed microphone designs do not allow arbitrary shaping of the directivity patterns, since their directivity pattern is entirely determined by the mechanical construction.
To partly overcome the problem, some specialized acoustical structures have been designed, which can be used to create narrower directional patterns than those of first-order microphones. For example, when a tube with holes in it is attached to an omnidirectional microphone, a microphone with narrow directional pattern can be created. These microphones are called shotgun or rifle microphones. However, they typically do not have a flat frequency response, that is, the directivity pattern is narrowed at the cost of the quality of the recorded sound. Furthermore, the directivity pattern is predetermined by the geometric construction and, thus, the directivity pattern of a recording performed with such a microphone cannot be controlled after the recording.
Therefore, other methods have been proposed to partly allow to alter the directivity pattern after the actual recording. Generally, this relies on the basic idea of recording sound with an array of omnidirectional or directional microphones and to apply signal processing afterwards. Various such techniques have been recently proposed. A fairly simple example is to record sound with two omnidirectional microphones, which are placed close to each other, and to subtract both signals from each other. This creates a virtual microphone signal having a directional pattern equivalent to a dipole.
In other, more sophisticated schemes, the microphone signals can also be delayed or filtered before summing them up. Using forming, a signal corresponding to a narrow beam is formed by filtering each microphone signal with a specially designed filter and summing the signals up after the filtering (filter-sum beam forming). However, these techniques are blind to the signal itself, that is, they are not aware of the direction of arrival of the sound. Thus, a predetermined directional pattern may be defined, which is independent of the actual presence of a sound source in the predetermined direction. Generally, estimation of the “direction of arrival” of sound is a task of its own.
Generally, numerous different spatial directional characteristics can be formed with the above techniques. However, forming arbitrary spatially selective sensitivity patterns (i.e. forming narrow directional patterns) involves a large number of microphones.
An alternative way to create multi-channel recordings is to locate a microphone close to each sound source (e.g. an instrument) to be recorded and recreate the spatial impression by controlling the levels of the close-up microphone signals in the final mix. However, such a system demands a large number of microphones and a lot of user-interaction in creating the final down-mix.
A method to overcome the above problem is DirAC, which may be used with different microphone systems and which is able to record sound for reproduction with arbitrary loudspeaker set ups. The purpose of DirAC is to reproduce the spatial impression of an existing acoustical environment as precisely as possible, using a multi-channel loudspeaker system having an arbitrary geometrical set up. Within the recording environment, the responses of the environment (which may be continuous recorded sound or impulse responses) are measured with an omnidirectional microphone (W) and with a set of microphones allowing to measure the direction of arrival of sound and the diffuseness of sound.
In the following paragraphs and within the application, the term “diffuseness” is to be understood as a measure for a non-directivity of sound. That is, sound arriving at the listening or recording position with equal strength from all directions, is maximally diffused. A common way of quantifying diffusion is to use diffuseness values from the interval [0, . . . , 1], wherein a value of 1 describes maximally diffused sound and a value of 0 describes perfectly directional sound, i.e. sound arriving from one clearly distinguishable direction only. One commonly known method of measuring the direction of arrival of sound is to apply 3 figure-of-eight microphones (X, Y, Z) aligned with Cartesian coordinate axes. Special microphones, so-called “B-Format microphones”, have been designed, which directly yield all desired responses. However, as mentioned above, the W, X, Y and Z signals may also be computed from a set of discrete omnidirectional microphones.
In DirAC analysis, a recorded sound signal is divided into frequency channels, which correspond to the frequency selectivity of human auditory perception. That is, the signal is, for example, processed by a filter bank or a Fourier-transform to divide the signal into numerous frequency channels, having a bandwidth adapted to the frequency selectivity of the human hearing. Then, the frequency band signals are analyzed to determine the direction of origin of sound and a diffuseness value for each frequency channel with a predetermined time resolution. This time resolution does not have be fixed and may, of course, be adapted to the recording environment. In DirAC, one or more audio channels are recorded or transmitted, together with the analyzed direction and diffuseness data.
In synthesis or decoding, the audio channels finally applied to the loudspeakers can be based on the omnidirectional channel W (recorded with a high quality due to the omnidirectional directivity pattern of the microphone used), or the sound for each loudspeaker may be computed as a weighted sum of W, X, Y and Z, thus forming a signal having a certain directional characteristic for each loudspeaker. Corresponding to the encoding, each audio channel is divided into frequency channels, which are optionally further divided into diffuse and non-diffuse streams, depending on analyzed diffuseness. If diffuseness has been measured to be high, a diffuse stream may be reproduced using a technique producing a diffuse perception of sound, such as the decorrelation techniques also used in Binaural Cue Coding.
Non-diffused sound is reproduced using a technique aiming to produce a point-like virtual audio source, located in the direction indicated by the direction data found in the analysis, i.e. the generation of the DirAC signal. That is, spatial reproduction is not tailored to one specific, “ideal” loudspeaker set-up, as in conventional techniques (e.g. 5.1). This is particularly the case, as, the origin of sound is determined as direction parameters (i.e. described by a vector) using the knowledge about the directivity patterns on the microphones used in the recording. As already discussed, the origin of sound in 3-dimensional space is parameterized in a frequency selective manner. As such, the directional impression may be reproduced with high quality for arbitrary loudspeaker set-ups, as far as the geometry of the loudspeaker set-up is known. DirAC is therefore, not limited to special loudspeaker geometries and generally allows for a more flexible spatial reproduction of sound.
DirAC, cf. Pulkki, V., Directional audio coding in spatial sound reproduction and stereo upmixing,” In Proceedings of The AES 28th International Conference, pp. 251-258, Pite{dot over (a)}, Sweden, Jun. 30-Jul. 2, 2006, provides a system for representing spatial audio signals based on one or more downmix signals plus additional side information. The side information describes, among other possible aspects, the direction of arrival of the sound field in the degree of its diffuseness in a number of frequency bands, as it is shown in FIG. 5.
FIG. 5 exemplifies a DirAC signal, which is composed of three directional components as, for example, figure-of-8 microphone signals X, Y, Z plus an omnidirectional signal W. Each of the signals is available in the frequency domain, which is illustrated in FIG. 5 by multiple stacked planes for each of the signals. Based on the four signals an estimation of a direction and a diffuseness can be carried out in blocks 510 and 520, which exemplify said estimation of the direction and the diffuseness for each of the frequency channels. The result of these estimations are given by the parameters θ(t,f), φ(t,f) and ψ(t,f) representing the azimuth angle, the elevation angle and the diffuseness for each of the frequency layers.
The DirAC parameterization can be used to easily implement a spatial filter with a desired spatial characteristic, for example only passing sound from the direction of a particular talker. This can be achieved by applying a direction/diffuseness and optionally frequency dependent weighting to the downmix signals as illustrated in FIGS. 6 and 7.
FIG. 6 shows a decoder 620 for reconstruction of an audio signal. The decoder 620 comprises a direction selector 622 and an audio processor 624. According to the example of FIG. 6 a multi-channel audio input 626 recorded by several microphones is analyzed by a direction analyzer 628 which derives direction parameters indicating a direction of origin of a portion of the audio channels, i.e. the direction of origin of the signal portion analyzed. The direction, from which most of the energy is incident to the microphone is chosen and the recording position is determined for each specific signal portion. This can, for example, be also done using the DirAC-microphone-techniques previously described. Other directional analysis methods based on recorded audio information may be used to implement the analysis. As a result, the direction analyzer 628 derives direction parameters 630, indicating the direction of origin of a portion of an audio channel or of the multi-channel signal 626. Furthermore, the directional analyzer 628 may be operative to derive a diffuseness parameter 632 for each signal portion, for example, for each frequency interval or for each time-frame of the signal.
The direction parameter 630 and, optionally, the diffuseness parameter 632 are transmitted to the direction selector 620, which is implemented to select a desired direction for origin with respect to a recording position or a reconstructed portion of the reconstructed audio signal. Information on the desired direction is transmitted to the audio processor 624. The audio processor 624 receives at least one audio channel 634, having a portion, for which the direction parameters have been derived. The at least one channel modified by audio processor may, for example, be a down-mix of the multi-channel signal 626, generated by conventional multi-channel down-mix algorithms. One extremely simple case would be the direct sum of the signals of the multi-channel audio input 626. However, as the concept is not limited by the number of input channels, all audio input channels 626 can be simultaneously processed by audio decoder 620.
The audio processor 624 modifies the audio portion for deriving the reconstructed portion of the reconstructed audio signal, wherein the modifying comprises increasing an intensity of a portion of the audio channel having direction parameters indicating a direction of origin close to the desired direction of origin with respect to another portion of the audio channel having direction parameters indicating a direction of origin further away from the desired direction of origin. In the example of FIG. 6, the modification is performed by multiplying a scaling factor 636 (q) with the portion of the audio channel to be modified. That is, if the portion of the audio channel is analyzed to be originating from a direction close to the selected desired direction, a large scaling factor 636 is multiplied with the audio portion. Thus, at its output 638, the audio processor outputs a reconstructed portion of the reconstructed audio signal corresponding to the portion of the audio channel provided at its input. As furthermore indicated by the dashed lines at the output 638 of the audio processor 624, this may not only be performed for a mono-output signal, but also for multi-channel output signals, for which the number of output channels is not fixed or predetermined.
In other words, the audio decoder 620 takes its input from such directional analysis as, for example, used in DirAC. Audio signals 626 from a microphone array may be divided into frequency bands according to the frequency resolution of the human auditory system. The direction of sound and, optionally, diffuseness of sound is analyzed depending on time at each frequency channel. These attributes are delivered further as, for example, direction angles azimuth (azi) and elevation (ele), and as diffuseness index (ψ), which varies between zero and one.
Then, the intended or selected directional characteristic is imposed on the acquired signals by using a weighting operation on them, which depends on the direction angles (azi and ele) and, optionally, on the diffuseness (T). Evidently, this weighting may be specified differently for different frequency bands, and will, in general, vary over time.
FIG. 7 shows a further example based on DirAC synthesis. In that sense, the example of FIG. 7 could be interpreted to be an enhancement of DirAC reproduction, which allows to control the level of the sound depending on analyzed direction. This makes it possible to emphasize sound coming from one or multiple directions, or to suppress sound from one or multiple directions. When applied in multi-channel reproduction, a post-processing of the reproduced sound image is achieved. If only one channel is used as output, the effect is equivalent to the use of a directional microphone with arbitrary directional patterns during recording of the signal. As shown in FIG. 7, the derivation of direction parameters, as well as the derivation of one transmitted audio channel is shown. The analysis is performed based on B-format microphone channels w, X, Y and Z, as, for example, recorded by a sound field microphone.
The processing is performed frame-wise. Therefore, the continuous audio signals are divided into frames, which are scaled by a windowing function to avoid discontinuities at the frame boundaries. The windowed signal frames are subjected to a Fourier transform in a Fourier transform block 740, dividing the microphone signals into N frequency bands. For the sake of simplicity, the processing of one arbitrary frequency band shall be described in the following paragraphs, as the remaining frequency bands are processed equivalently. The Fourier transform block 740 derives coefficients describing the strength of the frequency components present in each of the B-format microphone channels W, X, Y, and Z within the analyzed windowed frame. These frequency parameters 742 are input into audio encoder 744 for deriving an audio channel and associated direction parameters. In the example shown in FIG. 7, the transmitted audio channel is chosen to be the omnidirectional channel 746 having information on the signal from all directions. Based on the coefficients 742 for the omnidirectional and the directional portions of the B-format microphone channels, a directional and diffuseness analysis is performed by a direction analysis block 748.
The direction of origin of sound for the analyzed portion of the audio channel is transmitted to an audio decoder 750 for reconstructing the audio signal together with the omnidirectional channel 746. When diffuseness parameters 752 are present, the signal path is split into a non-diffuse path 754a and a diffuse path 754b. The non-diffuse path 754a is scaled according to the diffuseness parameter, such that, when the diffuseness ψ is low, most of the energy or of the amplitude will remain in the non-diffuse path. Conversely, when the diffuseness is high, most of the energy will be shifted to the diffuse path 754b. In the diffuse path 754b, the signal is decorrelated or diffused using decorrelators 756a or 756b. Decorrelation can be performed using conventionally known techniques, such as convolving with a white noise signal, wherein the white noise signal may differ from frequency channel to frequency channel. As long as decorrelation is energy preserving, a final output can be regenerated by simply adding the signals of the non-diffuse signal path 754a and the diffuse signal path 754b at the output, since the signals at the signal paths have already been scaled, as indicated by the diffuseness parameter ψ.
When the reconstruction is performed for a multi-channel set-up, the direct signal path 754a as well as the diffuse signal path 754b are split up into a number of sub-paths corresponding to the individual loudspeaker signals at split up positions 758a and 758b. To this end, the split up at the split up position 758a and 758b can be interpreted to be equivalent to an up-mixing of the at least one audio channel to multiple channels for a playback via a speaker system having multiple loudspeakers.
Therefore, each of the multiple channels has a channel portion of the audio channel 746. The direction of origin of individual audio portions is reconstructed by redirection block 760 which additionally increases or decreases the intensity or the amplitude of the channel portions corresponding to the loudspeakers used for playback. To this end, redirection block 760 generally relies on knowledge about the loudspeaker setup used for playback. The actual redistribution (redirection) and the derivation of the associated weighting factors can, for example, be implemented using techniques using as vector based amplitude panning. By supplying different geometric loudspeaker setups to the redistribution block 760, arbitrary configurations of playback loudspeakers can be used in embodiments, without a loss of reproduction quality. After the processing, multiple inverse Fourier transforms are performed on frequency domain signals by inverse Fourier transform blocks 762 to derive a time domain signal, which can be played back by the individual loudspeakers. Prior to the playback, an overlap and add technique is performed by summation units 764 to concatenate the individual audio frames to derive continuous time domain signals, ready to be played back by the loudspeakers.
According to the example shown in FIG. 7, the signal processing of DirAC is amended in that an audio processor 766 is introduced to modify the portion of the audio channel actually processed and which allows to increase an intensity of a portion of the audio channel having direction parameters indicating a direction of origin close to a desired direction. This is achieved by application of an additional weighting factor to the direct signal path. That is, if the frequency portion processed originates from the desired direction, the signal is emphasized by applying an additional gain to that specific signal portion. The application of the gain can be performed prior to the split point 758a, as the effect shall contribute to all channel portions equally.
The application of the additional weighting factor can be implemented within the redistribution block 760 which, in that case, applies redistribution gain factors increased by the additional weighting factor.
When using directional enhancement in reconstruction of a multi-channel signal, reproduction can, for example, be performed in the style of DirAC rendering, as shown in FIG. 7. The audio channel to be reproduced is divided into frequency bands equal to those used for the directional analysis. These frequency bands are then divided into streams, a diffuse and a non-diffuse stream. The diffuse stream is reproduced, for example, by applying the sound to each loudspeaker after convolution with 30 ms white noise bursts. The noise bursts are different for each loudspeaker. The non-diffuse stream is applied to the direction delivered from the directional analysis which is, of course, dependent on time. To achieve a directional perception in multi-channel loudspeaker systems, simple pair-wise or triplet-wise amplitude panning may be used. Furthermore, each frequency channel is multiplied by a gain factor or scaling factor, which depends on the analyzed direction. In general terms, a function can be specified, defining a desired directional pattern for reproduction. This can, for example, be only one single direction, which shall be emphasized. However, arbitrary directional patterns can be easily implemented in line with FIG. 7.
In the following approach, a further example is described as a list of processing steps. The list is based on the assumption that sound is recorded with a B-format microphone, and is then processed for listening with multi-channel or monophonic loudspeaker set-ups using DirAC style rendering or rendering supplying directional parameters, indicating the direction of origin of portions of the audio channel.
First, microphone signals can be divided into frequency bands and be analyzed in direction and, optionally, diffuseness at each band depending on frequency. As an example, direction may be parameterized by an azimuth and an elevation angle (azi, ele). Second, a function F can be specified, which describes the desired directional pattern. The function may have an arbitrary shape. It typically depends on direction. It may, furthermore, also depend on diffuseness, if diffuseness information is available. The function can be different for different frequencies and it may also be altered depending on time. At each frequency band, a directional factor q from the function F can be derived for each time instance, which is used for subsequent weighting (scaling) of the audio signal.
Third, the audio sample values can be multiplied with the q values of the directional factors corresponding to each time and frequency portion to form the output signal. This may be done in a time and/or a frequency domain representation. Furthermore, this processing may, for example, be implemented as a part of a DirAC rendering to any number of desired output channels.
As previously described, the result can be listened to using a multi-channel or a monophonic loudspeaker system. Recently, parametric techniques for the bitrate-efficient transmission/storage of audio scenes containing multiple audio objects have been proposed, e.g. Binaural Cue Coding (Type 1), cf. C. Faller and F. Baumgarte, “Binaural Cue Coding—Part II: Schemes and applications”, IEEF Trans. on Speech and Audio Proc., vol. 11, no. 6, November 2003, or Joint Source Coding, cf. C. Faller, “Parametric Joint-Coding of Audio Sources”, 120th AES Convention, Paris, 2006, Preprint 6752, and MPEG Spatial Audio Object Coding (SAOC), cf. J. Herre, S. Disch, J. Hilpert, O. Hellmuth: “From SAC to SAOC—Recent Developments in Parametric Coding of Spatial Audio”, 22nd Regional UK AES Conference, Cambridge, UK, April 2007, J. Engdeg{dot over (a)}rd, B. Resch, C. Falch, O. Hellmuth, J. Hilpert, A. Hölzer, L. Terentiev, J. Breebaart, J. Koppens, E. Schuijers and W. Oomen: “Spatial Audio Object Coding (SAOC)—The Upcoming MPEG Standard on Parametric Object Based Audio Coding”, 124th AES Convention, Amsterdam 2008, Preprint 7377).
These techniques aim at perceptually reconstructing the desired output audio scene rather than by a waveform match. FIG. 8 shows a system overview of such a system (here: MPEG SAOC). FIG. 8 shows an MPEG SAOC system overview. The system comprises an SAOC encoder 810, an SAOC decoder 820 and a renderer 830. The general processing can be carried out in a frequency selective way, where the processing defined in the following can be carried out in each of the individual frequency bands. The SAOC encoder is input with a number of (N) input audio object signals, which are downmixed as part of the SAOC encoder processing. The SAOC encoder 810 outputs the downmix signal and side information. The side information extracted by the SAOC encoder 810 represents the characteristics of the input audio objects. For MPEG SAOC, the object powered for all audio objects are the most significant components of the side information. In practice, instead of absolute object powers, relative powers, called object level differences (OLD), are transmitted. The coherence/correlation between pairs of objects are called interobject coherence (IOC) and can be used to describe the properties of the input audio objects further.
The downmix signal and the side information can be transmitted or stored. To this end, the downmix audio signal may be compressed using well-known perceptual audio coders, such as MPEG-1 layer 2 or 3, also known as MP3, MPEG advance audio coding (AAC) etc.
On the receiving end, the SAOC decoder 820 conceptually tries to restore the original object signals, to which it is also referred to as object separation, using the transmitted side information. These approximated object signals are then mixed into a target scene represented by M audio output channels using a rendering matrix, being applied by the renderer 830. Effectively, the separation of the object signals is never executed since both the separation step and the mixing step are combined into a single transcoding step, which results in an enormous reduction in computational complexity.
Such a scheme can be very efficient, both in terms of transmission bitrate, it only needs to transmit a few downmix channels plus some side information instead of N object audio signals plus rendering information or a discrete system, and computational complexity, the processing complexity relates mainly to the number of output channels rather than the number of audio objects. Further advantages for the user on the receiving end include the freedom of choosing a rendering setup of his/her choice, e.g. mono, stereo, surround, virtualized headphone playback etc. and the feature of user interactivity: The rendering matrix, and thus the output scene, can be set and changed interactively by the user according to will, personal preference or other criteria, e.g. locate the talkers from one group together in one spatial area to maximize discrimination from other remaining talkers. This interactivity is achieved by providing a decoder user interface.
A conventional transcoding concept for transcoding SAOC into MPEG surround (MPS) for multi channel rendering is considered in the following. Generally, the decoding of SAOC can be done by using a transcoding process. MPEG SAOC renders the target audio scene, which is composed of all single audio objects, to a multi-channel sound reproduction setup by transcoding it into the related MPEG surround format, cf. J. Herre, K. Kjörling, J. Breebaart, C. Faller, S. Disch, H. Purnhagen, J. Koppens, J. Hilpert, J. Rödén, W. Oomen, K. Linzmeier, K. S. Chong: “MPEG Surround—The ISO/MPEG Standard for Efficient and Compatible Multichannel Audio Coding”, 122nd AES Convention, Vienna, Austria, 2007, Preprint 7084.
According to FIG. 9, the SAOC side information is parsed 910 and then transcoded 920 together with user supplied data about the playback configuration and object rendering parameters. Additionally, the SAOC downmix parameters are conditioned by a downmix preprocessor 930. Both the processed downmix and the MPS side information can then be passed to the MPS decoder 940 for final rendering.
Conventional concepts have the disadvantage that they are either easy to implement as, for example, for the case of DirAC, but user information or user individual rendering cannot be applied, or they are more complex to implement, however, provide the advantage that user information can be considered as, for example, for SAOC.